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FOREWORD 


The  Army  Research  Institute  Field  Unit  at  Fort  Rucker,  Ala.,  pro- 
vides timely  support  to  the  U.S.  Army  Aviation  Center  (USAAVNC)  in  the 
area  of  aviation  training  research  and  development.  This  research  re- 
port documents  work  performed  as  a part  of  the  Field  Unit's  Flight  Simu- 
lation Work  Unit  research  efforts  in  designing  and  conducting  studies 
to  evaluate  helicopter  simulator  systems  and  subsystems  fob  training 
suitability. 

The  entire  program  of  aviation  training  research  and  development 
is  responsive  to  the  requirements  of  Army  Project  2Q763743A772 , Aircrew 
Performance  in  the  Tactical  Environment,  and  the  Directorate  of  Train- 
ing Developments,  USAAVNC,  Fort  Rucker,  Ala. 

CPT  Jack  Davis  of  DTD,  USAAVNC,  served  as  monitor  for  this  col- 
laborative effort  between  USAAVNC,  the  simulator  manufacturer,  the  ARI 
Field  Unit,  and  the  Project  Manager  for  Training  Devices.  The  instruc- 
tor pilots  from  the  Department  of  Undergraduate  Flight  Training,  USAAVNC, 
were  CW2  David  Broadnax,  CW2  Carol  Courtney,  CW2  Douglas  Joyce,  and 
CW2  Tom  Adkins.  CPT  Dana  Young,  DRTM,  USAAVNC,  assisted  in  coordinating 
efforts  to  obtain  subjects.  SFC  Floyd  Ingram  and  SSG  James  Parker 
served  as  UH1FS  console  operators,  and  Mr.  Mike  Stephenson  served  as 
programer . 
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EVALUATION  OF  THE  SINGER  NIGHT  VISUAL  SYSTEM  COMPUTER-GENEPATED  IMAGE 
DISPLAY  ATTACHED  TO  THE  UH-1  FLIGHT  SIMULATOR 


BRIEF 


Requirement : 

To  evaluate  the  suitability  of  the  Singer  night  visual  system 
computer-generated  image  display  for  integration  with  the  UH-1  Flight 
Simulator  for  training. 


Procedure : 

Three  evaluations  were  made  over  a 4-month  period.  In  the  first, 
four  qualified  contact  instructor  pilots  conducted  a formal  rating  of 
the  capability  of  the  system  to  accomplish  each  night  contact  maneuver 
contained  in  the  Flight  Training  Guide.  Next,  14  student  pilots  were 
trained  in  the  simulator  on  five  night  contact  maneuvers  and  performance 
was  compared  to  seven  control  student  pilots  in  a transfer  of  training 
study.  An  informal  evaluation,  using  instrument  instructors,  was  con- 
ducted to  determine  the  device's  capability  to  enhance  instrument  ap- 
proach to  landing. 

Findings : 

Instructor  pilots  rated  the  device  acceptable  for  training,  al- 
though there  were  notable  deficiencies  in  visual  cuing  as  well  as  in 
UH1FS  flight  characteristics  after  integration  of  the  NVS. 

There  was  no  transfer  effect,  measured  either  by  trials  to  cri- 
terion or  instructor  ratings  of  performance,  on  the  student  pilots. 

The  device  was  acceptable  as  an  adjunct  to  instrument  approach 
training. 

Utilization  of  Findings: 

Recommended  modifications  to  hardware  and  further  evaluations, 
sampling  pilot  performance  at  different  points  in  the  student  curricu- 
lum, are  necessary  before  judgment  can  be  conclusive  as  to  the  suita- 
bility of  the  night  visual  system  display  in  the  training  program. 
Therefore,  it  was  decided  not  to  procure  a night  visual  system  for 
the  UH1FS . 
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EVALUATION  OF  THE  SINGER  NIGHT  VISUAL  SYSTEM  COMPUTER-GENERATED  IMAGE 
DISPLAY  ATTACHED  TO  THE  UH-1  FLIGHT  SIMULATOR 


INTRODUCTION 

The  Army  has  increased  its  emphasis  on  synthetic  flight  training  in 
the  undergraduate  initial  entry  rotary  wing  (IERW)  course,  the  graduate 
training  and  transition  courses,  and  in  maintenance  of  instrument  profi- 
ciency of  its  rated  aviators.  Helicopter  simulators  under  development  are 
scheduled  to  have  complete  six-degree-of-freedom  motion  systems  and  camera 
model  board,  closed  circuit  television  visual  displays.  At  present,  the 
Army's  UH-1  Flight  Simulator  is  an  instruments-only  device  with  no  out-the- 
window  visual  display.  The  UH-1  Flight  Simulator  has  been  successful  from 
an  instrument  training  point  of  view,  and  it  was  deemed  desirable  to  inves- 
tigate the  possibility  of  increasing  the  device's  training  utility  by 
attaching  a visual  display  device  to  provide  for  night  contact  flight 
training . 

A Letter  Requirement  (LR)  was  written  by  the  U.S.  Army  Aviation  Center 
and  forwarded  to  the  Project  Manager  for  Training  Devices  (PM  TRADE) . Poten- 
tial sources  for  night  visual  displays  were  invited  to  provide  a visual 
system  for  evaluation  with  no  contractual  obligation  to  the  government  or 
the  manufacturer.  As  a result,  the  Singer  Company,  Simulation  Products 
Division,  agreed  to  provide  their  APO  Night  Visual  Calligraphic  Independent 
Research  and  Development  Product  Improvement  System  for  evaluation.  This 
report  documents  the  evaluation  of  that  system. 


Objective 

The  objective  of  the  evaluation  was  to  determine  the  suitability  of 
the  Singer  Night  Visual  System  (NVS)  computer-generated  image  display, 
which  was  integrated  with  the  UH-1  Flight  Simulator  for  training. 


Procedure 

Two  approaches  were  used  to  evaluate  the  NVS.  In  the  first  approach, 
four  assigned  instructor  pilots  were  given  thorough  orientations  and 
training  in  the  device.  Each  was  then  asked  to  complete  a comprehensive 
rating  of  the  device's  capabilities  to  accomplish  the  maneuvers  described 
in  the  Flight  Training  Guide.  The  second  approach  to  the  evaluation  was 
a transfer-of- training  study  using  IERW  student  pilots. 
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Results 


In  the  first  evaluation,  the  instructor  pilots  rated  the  device 
suitable  to  accomplish  most  of  the  maneuvers  listed  in  the  Flight  Training 
Guide . 

In  the  second  evaluation — the  transfer  study — comparisons  were  made 
of  the  number  of  trials  to  reach  criterion  performance  and  instructor 
ratings  of  performance  for  those  trained  in  the  simulator  and  those  who 
received  helicopter  training  in  five  selected  maneuvers.  There  was  no 
evidence  of  any  increase  in  skill  as  a function  of  simulator  training. 


Conclusions 


The  following  conclusions  were  reached,  based  on  the  two  evaluations. 

1.  Instructor  pilot  ratings  of  the  NVS/UH1FS  showed  the  system 
acceptable  for  further  training,  but  there  were  definite 
deficiencies  in  the  visual  display  scene  content  that  precluded 
activities  such  as  maintaining  ground  track  and  judging  distances 
and  altitudes. 

2.  Window  location  of  the  side  window  display,  coupled  with  a 
narrow  field  of  view,  required  an  adjustment  in  the  simulator 
training  and  altered  the  method  of  instruction  in  some  basic 
maneuvers . 

3.  Maneuvers  requiring  considerable  out-the-window  visual  search, 
such  as  the  night  approach,  were  more  difficult  to  accomplish 
and  yielded  greater  variability  than  maneuvers  in  which  outside 
referents  were  not  required. 

4.  Hardware  reliability  caused  evaluation  management  problems  at 
the  outset,  and  the  general  problems  attendant  on  integration 
of  a visual  display  with  an  instrument-only  flight  simulator 
were  formidable. 

5.  On  the  basis  of  the  data  collected,  the  tested  device  was  judged 
not  suitable  for  training  night  contact  helicopter  maneuvers. 
However,  before  a definitive  statement  is  made  concerning  the 
suitability  of  night  visual  system  displays  in  such  an  application, 
hardware  improvements  and  appropriate  sampling  of  students  in  a 
different  phase  of  training  are  recommended. 
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Recommendations 


The  following  recommendations  were  made,  based  on  the  evaluation  of 
the  NVS/UH1FS . 

1.  The  cockpit  environment  should  be  improved  by  eliminating  the 
black  light-shield  curtain,  devising  a door  seal  to  exclude  stray 
light,  and  improving  the  ventilation. 

2.  The  right  window  location  should  be  empirically  determined. 

3.  The  eye  height  should  be  adjusted  to  yield  a perceptually 
correct  perspective,  in  focus. 

4.  The  front  window  display  technique  should  be  further  explored 
to  yield  a display  that  could  rotate,  so  that  the  ground  could 
be  seen  on  approach. 

5 . Surface  texture  should  be  improved . 

6.  Scene  luminance  should  be  improved.  Techniques  such  as  brighten- 
ing the  entire  surface  of  the  display  should  be  explored. 

7.  Measurements  should  be  made  to  determine  whether  a lag  exists 
in  the  visual  system  relative  to  the  motion  system  and,  if  so, 
the  duration  of  the  lag. 


TECHNICAL  SUPPLEMENT 


BACKGROUND 

The  Army  has  increased  its  emphasis  on  synthetic  flight  training  as 
an  adjunct  to  regular  training  in  the  undergraduate  initial  entry  rotary 
wing  (IERW)  course,  the  graduate  training  and  transition  courses,  and  in 
maintenance  of  instrument  proficiency  of  its  rated  aviators.  Helicopter 
simulators  under  development  are  scheduled  to  have  complete  six-degree- 
of-freedom  motion  systems  and  camera  model  board,  closed  circuit  television 
visual  displays.  But  at  present,  the  Army's  UH-1  flight  simulator  is  an 
instruments-only  device  with  no  out-the-window  visual  display.  Because 
the  UH1FS  has  been  so  successful  from  an  instrument  training  point  of 
view,  it  was  deemed  desirable  to  increase  the  device's  capability,  if 
possible,  by  attaching  a visual,  out-the-window  display. 

Several  problems  arise  in  connection  with  the  addition  of  a visual 
display  to  the  UH1FS.  The  physical  hardware  limitations  of  the  UH1FS  and 
the  lack  of  space  in  the  building  in  which  the  UH1FS  is  housed  preclude 
the  attachment  of  a camera  model  board  visual  system.  Alternative  visual 
displays  are  point  light  source  and  transparency  systems,  open- loop  movies, 
semi-closed  loop  (VAMP)  movies,  computer-generated  image  (CGI)  displays, 
flying  spot  scanner- transparency  systems,  laser/holographic  displays,  and 
hybrid  systems  consisting  of  two  or  more  of  the  basic  approaches  (Smode, 
1972) . State-of-the-art  and  physical  limitations  ruled  out  the  basic 
approaches  with  the  exception  of  CGI  systems.  Day  color  CGI  has  been 
rejected  in  the  past  because  of  state-of-the-art,  particularly  for  terrain 
flight  levels.  But  night-only  calligraphic  CGI  display  systems  are  suffi- 
ciently advanced  that  it  was  plausible  to  consider  attachment  of  such  a 
system,  particularly  with  an  increased  emphasis  in  night  training  of  the 
Army  aviator.  In  addition,  physical  limitations  that  limited  the  use  of 
other  systems  did  not  seem  to  constrain  the  UH1FS  for  night-only  CGI. 
Therefore,  the  evaluation  of  such  a system  to  enhance  the  capability  of 
the  UH1FS  was  desirable. 

As  a result  of  the  considerations  cited  above,  a Letter  Requirement 
(LR)  for  a night  visual  system  (NVS)  was  written.  This  LR  justified  the 
evaluation  on  the  basis  of  the  increased  emphasis  on  night  training.  It 
also  specified  that  the  CGI  NVS  should  be  integrated  without  major  modifi- 
cation to  the  SFTS  UH1FS,  and  that  no  degradation  of  present  capabilities 
of  the  UH1FS  should  result. 
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The  LR  specified  the  principal  characteristics  of  the  device  in 
terms  of  the  number  of  windows,  number  of  light  points  available,  cycle 
rate,  and  acceptable  flicker.  In  addition,  the  LR  specified  that  the 
device  would  be  capable  of  presenting  runway  texture,  horizon  glow,  and 
moon  reflections.  The  LR  further  stated  that  the  device  was  expected  to 
increase  the  capabilities  of  the  UH1FS  in  the  following: 

1.  Hovering  and  ground  operations, 

2.  Effective  transition  from  hovering  flight  to  forward  and/or 
reverse  flight, 

3.  Night  autorotation  to  touchdown, 

4.  Night  takeoff  and  approaches  to  confined  areas,  pinnacles,  and 
stagefield  using  T-bar  lighting, 

5.  Capability  to  break  out  on  instrument  approaches  and  execute  a 
landing  to  Standard  A lighting. 

The  LR  listed  the  following  further  requirements  in  order  of  priority: 

1.  Confined  area  landing  and  takeoff  using  T-bar  lighting, 

2.  Stagefield  maneuvers  using  T-bar  and  normal  lighting, 

3.  Tactical  cross-country  flight  using  a minimum  number  of  light 
points  required  for  navigation  and/or  orientation, 

4.  Instrument  approaches  to  Standard  A lighting  with  visual  landing, 

5.  Pinnacle  takeoff  and  landing  using  T-bar  lighting, 

6.  Night  formation  flight, 

7.  Night  military  ground  handling  and  parking  procedures. 

As  a result  of  the  LR,  letters  were  sent  to  four  firms  involved  in 
CGI  night  visual  systems  as  a first  step  in  obtaining  demonstration  models 
for  evaluation.  The  letter  invited  the  firms  to  demonstrate  the  capabili- 
ties of  their  systems  on  the  UH1FS,  without  contractual  obligation  on  the 
part  of  the  government  or  the  contractor. 

The  potential  contractors  were  provided  with  the  capabilities  and 
priorities  established  in  the  LR,  a milestone  schedule,  and  data  handouts 
on  the  characteristics  of  the  UH1FS.  In  response  to  the  invitation,  the 
Singer  Company,  Simulation  Products  Division,  agreed  to  participate  in 
the  evaluation. 
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This  report  presents  the  results  of  the  evaluation  of  the  Singer 
Night  Visual  System  attached  to  the  UH1FS.  The  evaluation  was  conducted 
during  the  period  27  June  1976  through  1 October  1976. 


SIMULATOR  SUBSYSTEMS  DESCRIPTIONS 
Visual  Device  Description 

The  visual  system  evaluated  in  this  study  was  the  APO  Night  Visual 
Calligraphic  Independent  Research  and  Development  (IR&D)  Product  Improve- 
ment System,  a nomenclature  assigned  by  the  Singer  Company.  It  was  a 
three-window  display  with  two  forward  windows  centered  laterally  in  front 
of  the  pilot  and  copilot  and  extending  from  18.5°  below  the  eyepoint  to 
10°  above  the  eyepoint.  One  window  was  fixed  to  the  right  side  as  close 
to  the  front  window  as  possible,  and  extended  from  5°  above  the  horizon 
to  23.5°  below  the  horizon. 

The  system  was  originally  intended  to  simulate  five  flight  regimes: 
the  confined  landing  area,  stagefield  landing  area,  a tactical  cross- 
country flight,  instrument  approach  to  Standard  A lighting,  and  a pinnacle 
landing  area. 

System  specifications  for  the  confined  area  were  developed  by  Singer.'*' 
Some  modifications  to  these  specifications  were  required  and  were  incorpo- 
rated on  the  job. 

The  confined  landing  area  ground  surface  was  100  meters  square  and 
had  a greenish  ground  texture.  A vertical  surface  10  m high  joined  the 
ground  surface  along  the  upwind  edge  and  served  as  a barrier.  This  barrier 
surrounded  the  confined  landing  area,  and  the  horizon  was  not  visible  below 
10  m above  ground  level  (AGL) . The  upwind  and  downwind  barriers  were 
illuminated  and,  after  preliminary  trials,  it  was  necessary  to  imbed  several 
red  lights  at  varying  heights  above  the  ground  to  provide  altitude  cues  for 
landing  and  takeoff.  A Maltese  cross  was  also  placed  on  the  ground  surface 
to  provide  a drift-sense  cue. 

In  order  to  provide  visual  cues  for  rate  of  closure  to  the  confined 
landing  area,  it  was  necessary  to  add  three  light  panels,  in  a staggered 
sequence,  on  the  short-final  approach. 

The  configuration  for  the  stagefield  included  a landing  lane,  a 
parking  ramp,  and  lights.  A tower,  constructed  of  a series  of  light  chains, 
was  added  to  the  scene  to  provide  an  altitude  cue'  for  landing. 


^Systems  specifications  described  were  extracted  from  the  Systems 
Specifications  provided  by  Singer  Company. 


7 


The  landing  area  included  an  inverted  Y,  1 red  and  1 green  threshold, 
40  green  taxi  lights,  and  a green  and  white  rotating  beacon. 

Runway  texture  simulated  flat  concrete  with  6-in.  edge  lines,  a pad 
with  a white  60-ft.  Maltese  cro  at  each  end  of  the  runway,  and  a 2-ft. 
white  square  in  the  center  of  the  lane.  No  texture  appeared  outside  the 
runway  or  taxi  areas,  and  the  systems  called  for  all  surfaces  and  markings 
to  be  visible  at  least  30.5  m ahead  of  the  helicopter. 

The  specifications  also  called  for  a tactical  cross-country  flight 
representing  50  nautical  miles.  In  addition,  approximately  100  white 
lights  at  each  of  two  towns  near  the  edges  of  the  flight  area  and  an 
additional  1,800  white  and  red  lights  spread  randomly  over  the  remainder 
of  the  area  were  provided. 

A pinnacle  landing  area  was  constructed  with  a ground  surface  100  m 
x 100  m in  area.  This  flat  surface  was  30  m above  the  surrounding  ground 
level.  A vertical  surface  100  m wide  and  30  m high  was  placed  from  the 
approach  edge  of  the  landing  area  and  extended  downward  to  the  surround- 
ing terrain  level,  giving  the  appearance  of  a landing  shelf. 


UH1FS  Description 

The  flight  simulator  used  for  this  evaluation  was  the  UH1FS.  It  is 
a simulator  system  consisting  of  a complex  of  four  simulated  cockpits 
driven  by  a single  digital  computer  system.  Each  cockpit  is  mounted  on 
a f ive-degree-of-freedom  cascade  motion  base.  A central  instructor 
station  is  provided  from  which  training  performance  in  all  four  cockpits 
can  be  controlled  and  monitored.  The  UH1FS  is  designed  to  simulate  the 
cockpit  configuration  and  the  flight,  engine,  and  system  performance  of 
the  UH-1H  helicopter.  Each  trainee  station  consists  of  a cockpit  shell 
with  appropriate  equipment,  control  loading  system,  and  environmental 
effects  equipment,  including  the  cockpit  motion  system  and  sound  simula- 
tion equipment.  A problem  control  panel  and  trainee  information  display 
panel  are  also  included  in  each  cockpit. 


EVALUATION  PLAN 

The  requirements  and  priorities  established  in  the  LR  formed  the 
basis  for  the  design  of  the  evaluation.  However,  additional  available 
data  affected  the  approach  to  the  evaluation.  One  such  factor  was 
"Night  Hawk,"  and  it  played  an  important  role  in  developing  the  evalua- 
tion plan. 
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Night  Hawk  was  the  name  given  a series  of  exercises  conducted  at 
the  USAAVNC  to  determine  the  feasibility  of  night  training  in  all  heli- 
copter maneuvers,  to  include  emergency  procedures,  and  to  ascertain 
training  requirements  necessary  for  the  night  qualification  of  instructor 
pilots  in  these  operations.  Official  guidance  in  developing  the  plan  of 
evaluation  was  to  make  maximum  use  of  the  solutions  and  lessons  learned 
in  Night  Hawk. 

In  order  to  comply  with  the  guidance  concerning  Night  Hawk,  an 
experienced  Night  Hawk  instructor  pilot  (IP)  was  recruited  to  serve  as 
advisor  and  chief  pilot  during  the  planning  phase  of  the  evaluation. 

Since  the  lessons  learned  in  Night  Hawk  had  not  been  incorporated  in  the 
USAAVNC  IERW  curriculum,  this  IP  had  the  additional  responsibilities  of 
prescribing  the  maneuvers  and,  later,  of  developing  the  Flight  Training 
Guide  for  the  Night  Visual  System  evaluation. 


Approach 

In  general,  two  evaluations  of  the  Night  Visual  System  were  accom- 
plished. First,  the  instructor  pilots  were  given  a thorough  orientation 
and  training  in  the  device.  The  first  evaluation  required  instructor 
pilots  to  complete  a comprehensive  rating  of  the  device's  capabilities. 

The  second  evaluation  was  a transfer-of-training  study,  which  was  conducted 
using  IERW  students  in  a standard  transfer-of-training  paradigm. 


Evaluation  I:  Comprehensive  Instructor  Pilot  Rating 

Because  the  NVS  was  unique  to  the  Army's  training  experience,  it  was 
desirable  to  have  a broad  exposure  of  the  device  over  several  levels  of 
aviator  experience.  This  approach  would  assist  in  det'  onining  whether 
the  device  was  suitable  for  training  and  whether  it  r ght  be  more  appro- 
priate to  one  level  of  experience  than  to  another.  Therefore,  the  first 
evaluation  was  designed  to  consider  the  NVS/UH1FS  as  a system  and  to 
determine  whether  or  not  it  was  capable  of  providing  suitable  simulation 
to  accomplish  the  maneuvers  specified  in  the  Flight  Training  Guide. 


Method 

Evaluators . In  order  to  determine  the  system's  capability  of 
accomplishing  the  maneuvers,  four  Night  Hawk-qualified  instructor  pilots 
were  assigned  to  the  project.  Each  had  extensive  current  experience  as 
an  instructor,  and  one  was  a Standardization  Instructor  Pilot.  All  were, 
therefore,  thoroughly  familiar  with  the  requirements  of  Night  Hawk  and 
equally  conversant  with  accepted  instructor  methods. 
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The  instructor  pilots  had  to  be  trained  to  use  the  UH-1  flight  simu- 
lator as  an  instructional  tool.  In  addition,  they  had  to  learn  how  to  fly 
the  Night  Visual  Systero/UHIFS . The  instructor  pilots  worked  with  the  con- 
sole operators  to  learn  the  simulator  Initial  Condition  Sets  for  the  device 
and  how  to  use  the  device  as  an  ir  truction  tool.  In  addition,  the  IP's 
had  to  attain  a personal  proficier  / on  all  of  the  maneuvers.  These  IP's 
were  required  to  accomplish  the  tasks  with  and  without  motion. 

A special  student  evaluation  check  sheet  was  developed  for  this  effort, 
and  the  instructor  pilots  were  required  to  spend  time  in  the  simulator 
learning  how  to  use  the  device  and  how  to  grade  maneuvers  with  the  pilot 
evaluation  sheets.  In  addition,  informal  classes  were  conducted  in  grading 
procedures.  A minimum  of  5 hours  of  instruction  and  practice  was  used  by 
each  of  the  pilots  to  attain  a personal  proficiency  on  all  of  the  maneuvers. 

Flight  Training  Guide.  The  Flight  Training  Guide  is  a standard  docu- 
ment for  each  course  of  flight  instruction  at  the  USAAVNC . It  is  divided 
into  four  sections  and  states  in  detail  the  flight  training  maneuvers  and 
procedures  to  be  taught  in  the  course.  For  each  maneuver  the  training 
objective  is  specified,  the  maneuver  requirements  are  stated,  the  maneuver 
is  analyzed,  and  some  common  errors  are  listed.  Because  the  Guide  is 
complete  for  the  particular  course  of  instruction,  this  evaluation  required 
each  IP  to  conduct  an  independent  rating  of  the  system  on  each  maneuver  in 
the  Guide. 

Rating  Scale.  An  8-point  rating  scale  was  devised  for  use  by  the  IP's 
in  rating  the  device.  (A  copy  of  the  scale  appears  in  appendix  A.)  The 
scale  lists  a maneuver  by  name  and  then  expresses  each  of  the  maneuver 
requirements  in  terms  of  the  best  performance  possible.  Given  this  sentence 
stem,  which  describes  the  best  possible  performance,  the  IP  was  asked  to 
determine  how  that  stem  described  his  performance  of  that  maneuver  in  the 
simulator  and  to  place  a mark  in  the  block  that  corresponded  to  the  degree 
to  which  the  stem  described  his  performance. 

Initially,  the  plan  of  test  called  for  a study  with  and  without  motion. 
Two  IP's  rated  the  simulator  on  each  maneuver  with  motion  on,  then  made  a 
second  rating  with  motion  off.  The  other  two  IP's  followed  the  same  proce- 
dure first  with  motion  off,  followed  by  motion  on. 

Results  and  Discussion 

The  system  was  judged  to  be  suitable  for  the  transfer  experiment  and 
thus  for  training.  But  some  points  in  the  data  merit  discussion. 

The  Flight  Training  Guide  was  prepared  to  instruct  contact  flight 
maneuvers  and  was  applied  to  the  UH1FS/NVS  system,  as  previously  discussed. 
Because  the  main  interest  was  to  determine  whether  the  device  could  be  used 
to  train  night  contact  maneuvers,  no  attempt  was  made  to  evaluate  the  NVS 
separate  from  the  system. 
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The  stems  that  described  the  maneuver  elements  were  assumed  to  be 
of  equal  weight  in  contributing  to  the  maneuver  they  describe.  There- 
fore, to  obtain  an  IP  rating  of  each  maneuver,  IP  ratings  of  the  elements 
of  that  maneuver  were  averaged.  The  original  plan  for  the  transfer-of- 
training  test  called  for  instruction  with  motion  on  and  motion  off,  and 
the  IP's  rated  the  system  under  both  conditions.  However,  circumstances 
prevented  the  test  under  both  conditions  and  the  motion-on  condition  was 
used  in  the  study.  Consequently , data  are  reported  for  the  motion-on 
condition  only. 

Table  1 presents  the  results  of  the  instructors'  ratings  of  each  of 
the  maneuvers  on  the  8-point  rating  scale. 

The  maneuvers  were  also  ranked  on  the  basis  of  the  ratings.  Ranks 
were  computed  for  each  rater,  based  on  his  score  of  the  maneuver.  These 
ranks  were  then  summed  for  all  raters. 

The  rank  order  in  Table  2 shows  the  relative  rank,  with  those  ma- 
neuvers rated  highest  listed  first  and  others  following  in  a descending 
order  of  rank. 

To  determine  the  reliability  of  the  ratings  and  the  degree  of  accord 
among  the  raters,  two  statistics  were  computed:  Kendall's  coefficient  of 
concordance  (W)  , and  the  intraclass  correlation  (Seigel,  1956;  Guilford, 
1964)  . 

Kendall's  W was  used  to  test  whether  the  raters  were  using  the  same 
standards  in  ranking  the  maneuvers.  The  basic  formula  also  included  a 
correction  for  time.  A W of  .620  was  obtained,  yielding  an  obtained  Chi 
Square  of  49.602,  which  was  statistically  significant  (X22o  P 11  -001). 

These  results  are  interpreted  to  mean  that  the  raters  were  applying 
essentially  the  same  standards  in  ranking  the  21  maneuvers.  It  does  not 
mean  that  the  ordering  is  "correct"  in  an  absolute  sense.  However,  accord- 
ing to  Seigel,  the  best  estimate  of  the  true  ranks  is  the  ordering  of  the 
sums  of  the  ranks. 

In  order  to  determine  the  rater  reliability,  a two-way  analysis  of 
variance  was  computed  on  the  ratings,  and  the  intraclass  correlation  was 
computed  from  the  analysis.  Table  3 presents  the  summary  of  the  analysis. 

The  results  of  the  analysis  of  variance  indicated  statistically 
significant  differences  between  raters  and  between  maneuvers.  These 
results  are  not  surprising  and  can  be  taken  to  indicate  that  the  raters 
did  discriminate  between  maneuvers.  In  addition,  one  of  the  raters  did 
tend  to  rate  each  maneuver  lower  than  the  others  but  he  showed  the  same 
general  trend  in  his  rating  of  any  given  maneuver. 


Table  1 

Average  of  Maneuver  Element  Ratings 


INSTRUCTOR  RATING  (8-POINT  SCALE) 
MANEUVER  INSTRUCTOR:  A B C D 


Hover  Check 
Takeoff  to  Hover 
Go-No-Go  Procedures 
Landing  from  Hover 
Hovering  Turns 
Sideward  Hovering  Flight 
Rearward  Hovering  Flight 
Night  Takeoff 
Straight  & Level  Flight 
Level  Turns 

Climbing  and  Descending  Turns 

Decelerations 

Night  Approach 

Shallow  Approach 

Simulated  Hydraulic  Failure 

Night  Standard  Autorotation 

Night  Low  Level  Autorotation 

\ 

Night  Reconnaissance 
Night  Circling  Approach 
Night  Confined  Area 
Night  Pinnacle 


3.00 

6.67 

8.00 

5.00 

5. 00 

8.00 

8.00 

7.33 

4.00 

7.33 

8.00 

7.67 

7.00 

8.00 

8.00 

8.00 

4.00 

8.00 

8.00 

6.30 

4.67 

8.00 

8.00 

7.00 

1.00 

1.00 

1.00 

1.00 

5.4 

7.00 

8.00 

8.00 

8.00 

6.67 

6.00 

8.00 

6.00 

8.00 

7.00 

8.00 

6.00 

8.00 

7.00 

8.00 

7.00 

8.00 

8.00 

8.00 

5.75 

5.  75 

5.25 

7.75 

6.00 

6.75 

5.75 

7.75 

1.00 

7.29 

6.14 

8.00 

6.  75 

6.75 

6.5 

8.00 

6.50 

6.00 

5.25 

7.75 

3.40 

3.40 

3.60 

6.40 

6.00 

T.  25 

7.25 

6.50 

3.75 

4.50 

5.00 

5.50 

1.00 

1.00 

3.25 

2.00 

Table  2 

Rank  Order  of  Maneuvers  Based  on  IP  Ratings 


MANEUVER 

RANK 

SUM  OF 

RANKS 

Landing  from  Hover 

1.5*  (Tie) 

72.50 

Deceleration 

1.5* 

72.50 

Level  Turns 

3.5* 

61.50 

Climbing  and  Descending  Turns 

3.5*  (Tie) 

61.50 

Night  Takeoff 

5 

57.00 

Night  Standard  Autorotation 

6 

55.00 

Takeoff  to  Hover 

7 

54.50 

Straight  & Level  Flight 

8 

54.00 

Sideward  Hovering  Flight 

9 

52.50 

Go-No-Go  Procedure 

10 

49.00 

Hovering  Turns 

11 

48.00 

Night  Circling  Approach 

12 

46.50 

Shallow  Approach 

13 

43.00 

Simulated  Hydraulic  Power  Failure 

14 

41.50 

Night  Low  Level  Autnrotation 

15 

40.50 

Night  Approach 

16 

34.50 

Hover  Check 

17 

32.00 

Night  Confined  Area 

18 

18.00 

Night  Reconnaissance 

19 

17.00 

Night  Pinnacle 

20 

7.50 

Rearward  Hovering  Flight 


21 


5.50 


Table  3 

Analysis  of  Variance  for  Ratings 


Source 

SS 

df 

MS 

F 

P 

Raters 

45.94 

3 

15.31 

12.25 

<.01 

Maneuvers 

253.22 

20 

12.66 

10.13 

<•01 

Residual 

75.24 

60 

1.25 

Total 

374.40 

83 

The  computed  intraclass  correlation  was  .87.  This  result  indicates 
that  the  average  of  the  intercorrelations  of  the  ratings  is  .87.  Ac- 
cording to  Guilford  (1964) , if  the  intercorrelation  of  the  raters  is 
taken  to  be  a reliability  of  ratings,  then  the  typical  reliability  of  a 
single  rater's  ratings  is  on  the  order  of  .87. 

On  the  basis  of  the  data  it  was  concluded  that  the  instructors  were 
in  agreement  concerning  the  system’s  capabilities  and  that  their  ratings 
were  reliable.  The  system  was  judged  suitable  for  continuation  of  the 
evaluation.  The  raters'  opinions  of  the  visual  system's  capabilities 
and  limitations  are  further  explored  in  the  general  discussion  section 
of  this  report. 


Evaluation  II:  IERW  Transfer  of  Training 

The  method  chosen  to  determine  the  suitability  of  the  Night  Visual 
System  for  training  was  a transfer-of-training  study  using  IERW  students. 

Objectives . The  original  test  plan  specified  five  objectives: 


1.  To  compare  transfer  performance  of  IERW  students  trained  with 

visual  and  motion  simulation  of  night  maneuvers  with  performance 
of  those  trained  in  actual  aircraft; 


2.  To  compare  transfer  performance  of  IERW  students  trained  on 
night  maneuvers  in  the  UH1FS  using  automated  instruction  with 
that  of  IERW  students  trained  in  the  UHlFS  by  instructor  pilots; 

3.  To  compare  transfer  performance  of  rated  aviators  not  in  flying 
assignments  who  are  trained  with  visual  and  motion  simulation 

of  night  maneuvers  with  performance  of  those  trained  in  aircraft; 
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4.  To  compare  transfer  performance  of  rated  aviators  not  in  flying 
assignments  who  are  trained  with  visual  and  motion  simulation 
using  automated  instruction  with  performance  of  those  trained 

in  the  simulator  by  instructor  pilots,  and  to  compare  performance 
of  both  groups  with  performance  of  those  trained  in  the  aircraft; 

5.  To  obtain  quality  assurance  data  on  the  NVS . 

Two  factors  caused  the  original  objectives  to  be  revised  for  this 
evaluation:  (a)  a shortage  of  qualified  instructor  pilots  precluded  the 

testing  of  the  rated  aviators;  and  (b)  computer  rounding  techniques  pre- 
vented the  exact  replication  of  a maneuver,  and  as  a consequence,  the 
automated  demonstration  tapes  could  not  be  constructed.  Therefore,  the 
objectives  for  this  evaluation  were  limited  to  objectives  1 and  5 above. 


Method 

Subjects . Subject  student  pilots  were  volunteers  from  Officers  and 
Warrant  Officer  Candidates  (WOC's)  in  the  resident  IERW  classes.  Each 
class  is  divided  into  a section  of  officers  and  a section  of  WOC's.  In 
any  given  week,  the  officers  may  fly  in  the  morning  and  receive  academic 
training  in  the  afternoon,  while  the  WOC's  receive  academics  in  the  morn- 
ing and  flight  instruction  in  the  afternoon.  The  following  week  the 
schedule  is  reversed.  To  select  subjects,  a roster  of  officers  and 
WOC's  was  obtained  for  each  class,  10  names  were  selected  from  each  roster 
via  a table  of  random  numbers,  and  these  individuals  were  assembled 
according  to  section.  The  test  was  explained,  and  8 volunteers  were 
obtained  from  the  pool  of  10  subjects. 

Volunteers  were  necessary  because  the  curriculum  flow  did  not  allow 
time  to  conduct  this  study.  The  desirability  of  obtaining  students  prior 
to  their  exposure  to  night  training  made  the  last  2 weeks  of  instrument 
qualification  the  best  available  time.  They  had  completed  an  academic 
block  and  were  flying  only  half-days.  Because  checkrides  are  usually 
given  during  this  2-week  period,  many  subjects  would  have  considerable 
free  time  if  they  had  completed  academics  and  had  had  a checkride  the 
first  week.  In  some  cases  a subject  was  scheduled  for  a checkride  in 
the  second  week  and  was  not  available  for  the  evaluation.  Since  there 
was  the  possibility  of  disrupting  a student's  training  activity  but  there 
was  no  provision  for  any  type  of  remedial  training,  the  Department  of 
Resident  Training  Management  ( DRTM)  felt  it  would  not  be  expedient  to 
assign  students  to  serve  as  subjects. 

Complete  data  were  obtained  on  21  subjects,  and  partial  data  were 
collected  on  a larger  number.  Equipment  malfunction,  weather,  and  admin- 
istration difficulties  prevented  collection  of  a complete  set  of  data  on 
all  subjects. 
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Performance  Assessment.  The  standard  grade  sheet  used  in  the  courses 
of  instruction  was  deemed  unacceptable  for  this  study.  The  preferred 
method  was  to  grade  performance  as  objectively  as  possible  and  to  exclude 
categories  that  called  for  subjective  judgments. 

The  first  step  in  developing  che  grade  sheet  was  to  identify  each 
maneuver  to  be  trained  in  the  simulator  and  the  aircraft.  Next  came 
identification  of  maneuver  elements  in  each  maneuver  that  were  critical 
to  that  maneuver.  Most  of  this  information  is  specified  in  the  Flight 
Training  Guide  and  it  was  therefore  the  major  source  for  such  information. 

After  identifying  the  maneuver  and  its  elements,  an  index  of  desired 
performance  was  specified  for  each  element;  where  feasible  a band  of 
acceptable  performance  around  that  index  was  identified.  That  is, 
some  maneuvers  were  scored  in  a dichotomous  fashion,  e.g.,  yes  or  no, 
while  others  received  a numerical  score. 

Next,  for  those  elements  where  number  assignment  could  be  done,  the 
index  of  desired  performance,  identified  as  "correct"  performance,  was 
assigned  the  number  3.  For  example,  if  the  student  flew  the  correct 
altitude  on  a given  trial,  he  received  a 3 for  altitude.  If  altitude 
was  low  but  within  acceptable  limits,  a score  of  2 was  assigned.  If 
altitude  was  low  and  outside  the  limits,  the  score  of  1 was  assigned. 
Likewise,  high,  acceptable  performance  was  assigned  a 4 and  high,  unaccept- 
able performance  earned  a 5. 

For  all  maneuver  elements,  a subject  would  receive  either  a number 
or  Y for  yes,  N for  no,  R for  rough,  or  S for  smooth. 

In  addition  to  the  above  scale,  the  IP  was  also  required  to  rate 
the  student's  last  simulator  flight,  first  UH-1  flight,  and  last  UH-1 
flight,  using  a 12-point  scale  adapted  from  Reid  (1975) . 

The  performance  assessment  grade  slip  was  used  to  determine  both 
the  number  of  trials  to  reach  criterion  performance  and  the  rating  of 
performance  at  that  criterion  point. 

Each  IP  was  given  a draft  form  of  this  grading  slip  for  trial.  He 
was  requested  to  rate  performance  after  each  trial  of  each  element  of  the 
maneuver.  After  the  trial  period,  a conference  was  held  with  the  IP's 
to  discuss  refinement  of  the  grade  slip.  The  IP's  then  identified,  on 
each  maneuver,  the  element  or  elements  critical  to  the  maneuver.  "Critical" 
was  operationally  defined  as  that  element  that  can  be  identified  as  criti- 
cal to  the  success  or  failure  of  the  maneuver.  The  pilots  were  successful 
in  paring  the  list  of  elements  to  those  they  thought  must  be  done  correctly, 
within  limits,  in  order  to  execute  the  maneuver  successfully.  (The  grade 
sheet  appears  in  appendix  B.) 
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Maneuvers  Tested.  The  maneuvers  selected  for  inclusion  in  the  NVS 
evaluation  were  extracted  from  a list  of  maneuvers  recommended  for  a 12- 
hour  night  transition  course  for  IERW  students.  Additional  selection 
factors  were  the  capability  to  perform  the  maneuver  in  the  simulator  and 
in  the  aircraft,  the  time  available,  and  the  availability  of  an  adequate 
number  of  instructor  pilots.  Based  on  these  factors,  the  following  list 
of  maneuvers  was  selected  for  evaluation  in  this  study. 

1.  Night  takeoff  and  climb, 

2.  Night  cruise, 

3.  Night  approach  to  stagefield  to  an  inverted  Y, 

4.  Night  approach  to  a confined  area  (simulator  only)  (a  night 
circling  approach  to  an  inverted  Y was  used  as  a substitute 
for  the  confined  area  in  flight  tests) , 

5.  Night  autorotation, 

6.  Instrument  approach  and  breakout  to  landing  to  Standard  A 
lighting  (instrument  IP's  only). 

Experimental  Design.  The  diagram  below  depicts  the  final  experi- 
mental design  of  this  test. 


Method  of 

Flight 

Vehicle 

instruction 

sample 

UH-1  fit  simulator 

Instructor  pilot 

IERW 

UH-1H  helicopter 

Instructor  pilot 

IERW 

Procedure . Six  of  the  eight  subjects  selected  from  each  section 
were  used  while  two  were  alternates.  Four  of  the  six  received  simulator 
training  while  the  other  two  were  the  controls,  receiving  helicopter 
training  only.  Because  two  sections  comprise  a class,  two  students  from 
a section  would  receive  simulator  training  in  the  morning,  and  two  from 
the  other  section  would  receive  it  in  the  afternoon.  Thus,  four  students 
were  trained  each  week  in  the  simulator;  these  were  tested,  along  with  the 
control  subjects,  in  the  transfer  task. 

One  instructor  pilot  was  responsible  for  instructing  one  student  in 
the  simulator  and  then  for  instructing  two  students  in  the  aircraft — one 
previously  trained  in  the  simulator  and  one  not  previously  trained.  The 
IP  was  requested  to  instruct  a different  student  in  the  aircraft  from  the 
one  he  had  in  the  simulator. 
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Simulator  Instruction.  Simulator  instruction  was  on  a proficiency 
advancement  basis.  Instructional  periods  for  the  simulator  were  of 
approximately  1 hour  and  20  minutes  duration,  beginning  on  Monday  of 
each  week.  Upon  reporting,  a student  would  receive  a brief  orientation 
and  move  to  the  simulator.  Actual  instruction  was  in  the  sequence 
indicated  on  the  grade  sheet  and  was  on  a proficiency  basis.  Competence 
in  a maneuver  was  operationally  defined  as  two  successive  repetitions 
of  a maneuver  with  a score  of  2,  3,  or  4,  that  is,  within  tolerances. 
Unique  features  of  the  simulator  were  used  to  expedite  the  training. 
Rather  than  teach  an  entire  traffic  pattern  as  is  done  in  the  helicopter, 
the  IP's  were  requested  to  use  Initial  Condition  Sets.  Thus,  in  training 
night  takeoff  and  climb,  the  helicopter  was  established  on  the  lane  and 
the  maneuver  was  executed.  Upon  reaching  a climbing  attitude  on  altitude 
to  turn  downwind,  the  IP  would  freeze  the  device,  return  to  the  predesig- 
nated Initial  Condition  Set  on  the  lane  and  practice  the  takeoff  again. 
Students  would  then  progress  to  the  next  graded  maneuver  upon  reaching 
criterion.  This  served  to  save  time  and  concentrate  the  learning 
experience  on  a single  task  at  a time . 

When  a student  reached  criterion,  the  IP  would  usually  fly  some 
complete  maneuver  patterns  such  as  the  traffic  pattern.  Then  the 
student's  performance  was  rated  on  the  12-point  performance  scale. 

On  Thursday  and  Friday  nights  of  the  same  week  in  which  simulator 
training  was  accomplished,  the  simulator  and  aircraft-only  students 
reported  to  the  helicopter  for  IP  assignment  and  instruction  in  the 
aircraft. 

A stagefield,  reserved  for  the  training,  was  set  up  with  appro- 
priate air  traffic  control  personnel  and  with  minimum  lighting  to 
include  the  inverted  Y. 

In  order  to  establish  a baseline  for  performance  and  to  determine 
the  effect  of  prior  simulator  training,  the  IP's  were  requested  to 
demonstrate  a maneuver  once  and  then  to  allow  the  student  to  try  the 
maneuver  with  no  additional  instruction.  The  IP  scored  the  maneuver 
and  rated  the  performance  on  the  12-point  scale.  Then  the  student  was 
given  additional  instruction  as  necessary  to  reach  criterion.  The 
flight  on  which  criterion  was  reached  was  rated  on  the  12-point  scale. 


Results  and  Discussion 


On  the  basis  of  the  data  collected,  the  system  did  not  demonstrate 
transfer  of  training  when  used  on  the  samples  described.  These  data  are 
discussed  briefly  in  this  section  and  in  greater  detail  in  the  general 
discussion  section. 
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The  two  basic  forms  of  data  collected  during  the  evaluation  were 
trials  to  criterion  and  instructor  ratings  of  pilot  performance.  Dur- 
ing the  graded  training  sessions  the  IP  recorded  the  number  of  trials 
required  to  reach  the  criterion  of  two  successive  trials  with  all  ma- 
neuver elements  within  acceptable  limits,  and  he  also  rated  the  student's 
performance  on  the  12-point  rating  scale. 

Trials  to  Criterion.  The  data  were  analyzed  with  a Randomized 
Block  analysis  of  variance  as  described  by  Kirk  (1968).  Cochran's 
test  was  computed  and  the  data  were  heterogeneous.  Therefore,  a recip- 
rocal transformation  was  applied  to  the  data.  Table  4 presents  a sum- 
mary of  the  analysis  of  these  data. 


Table  4 

Analysis  of  Variance  of  Simulator  Trials  to  Criterion 


Source 

SS 

df 

MS 

F 

P 

Maneuvers 

.1102 

4 

.0276 

7.6667 

<•01 

Subjects 

.1285 

13 

.0099 

2.7500 

A 

O 

h-* 

Residual 

.1824 

51 

.0036 

Total 

.4211 

68 

One  score  in  the  data  was  missing,  and  that  datum  point  was  esti- 
mated using  accepted  statistical  technique.  One  degree  of  freedom  was 
subtracted  from  the  residual  error  term  and  from  the  total  degrees  of 
freedom  as  a result. 

The  data  showed  that  there  were  significant  differences  between  the 
maneuvers,  indicating  that  it  was  easier  to  reach  a criterion  level  of 
performance  on  one  or  more  maneuvers.  Tukey's  HSD  statistic  was  calcu- 
lated to  make  pair-wise  comparisons  among  the  means.  It  was  found  that 
the  mean  trials  to  criterion  for  the  night  approach  and  the  night  auto- 
rotation were  significantly  larger  than  for  the  takeoff.  In  addition, 
learning  to  fly  the  cruise  maneuver  required  significantly  fewer  trials 
than  the  night  approach,  night  autorotation,  or  night  circling  approach. 

Table  5 contains  the  mean  scores  for  maneuvers.  The  data  analyzed 
were  reciprocals,  but  the  means  in  Table  5 are  those  computed  from  the 
original  data  and  are  thus  presented  to  avoid  confusion  in  interpretation. 
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Table  5 

Mean  Trials  to  Criterion  for  Simulator-Trained  Students 


Maneuver 

Takeoff 
and  climb 

Cruise 

Night 

approach 

Night 

autorotation 

Night 
circling 
approach 
(confined  area) 

3.21 

2.64 

5.71 

5.64 

3.93 

There  were  also  significant  performance  differences  among  the  student 
pilots.  However,  this  result  was  interpreted  as  indicating  individual 
differences  in  learning  rates  and  was  not  pursued  with  further  analysis. 

A major  determiner  of  the  transfer  effect  was  the  comparison  of  the 
number  of  trials  the  helicopter- trained  group  required  to  reach  criterion 
performance  with  the  number  of  trials  required  by  those  who  were  trained 
in  the  simulator  first. 

1A  split-plot  factorial  design  analysis  of  variance  was  computed  on 

the  data.  The  Cochran  test  for  homogeneity  of  variance  was  not  signifi- 
cant. The  analysis  of  variance  least-squares  solution  was  computed  on 
the  data  because  of  planned  unequal  size  subgroups.  Table  6 presents 
a summary  of  the  analysis. 

Table  6 

Analysis  of  Variance  of  Aircraft  Trials  to  Criterion 


Source 

SS 

df 

MS 

F 

P 

Between  subjects 

31.83 

20 

Type  of  Training 

1.55 

1 

1.55 

.97 

NS 

Subjects  Within  Groups 

30.28 

19 

1.59 

Within  Subjects 

82.80 

84 

Maneuvers 

Type  Training 

8.62 

4 

2.16 

2.35 

NS 

x Maneuver 

3.89 

4 

.97 

1.05 

NS 

B x Subjects  Within  Groups 

70.29 

76 

.92 

Total 

114.63 

104 

20 


The  results  of  this  analysis  indicate  no  significant  differences 
between  those  trained  in  the  simulator  and  those  trained  in  the  air- 
craft only.  In  addition,  there  was  no  significant  difference  between 
maneuvers  in  terms  of  trials  to  criterion. 

It  is  also  informative  to  look  at  the  variability  in  each  of  the 
maneuvers.  Table  7 shows  the  variability  in  trials  to  criterion  per- 
formance on  maneuvers  performed  by  those  trained  in  the  aircraft  and 
those  trained  in  the  simulator. 


Table  7 


Trials  to  Criterion  Variance  of  Maneuvers  in  Helicopter 


Maneuver 


Takeoff 
& Climb 

Cruise 

Night 
Appro ach 

Night 

Autorotation 

Night 

Circling 

Approach 

Helicopter 
Training  Oily 

.24 

1.56 

.90 

.40 

1.23 

Simulator 

Training 

1.14 

.55 

1.49 

2.10 

.42 

Although  there  were  no  significant  differences  among  these  variances, 
Table  7 does  point  to  some  peculiar  patterns  in  the  distribution  of  the 
variances,  particularly  in  the  night  approach  and  the  night  autorotation. 
An  explanation  of  these  results  is,  at  best,  conjecture. 

Instructor  Ratings.  In  assessing  performance  it  was  reasoned  that 
the  case  could  arise  in  which  there  was  no  statistical  difference  in 
trials  to  criterion,  yet  the  instructor  could  determine  that  there  were 
qualitative  differences  which,  if  quantified,  could  indicate  significant 
performance  differences.  To  determine  this,  the  ratings  assigned  by 
the  IP's  on  the  first  flight  in  the  aircraft  were  analyzed.  A split- 
plot  factorial  design  with  a least-squares  solution  was  used  in  the 
analysis.  Cochran's  test  for  homogeneity  indicated  that  the  assumption 
of  homogeneity  could  not  be  accepted.  A square-root  transformation  of 
the  original  scores  was  applied  to  the  data.  Table  8 presents  a summary 
of  the  analysis  of  variance  of  ratings. 

The  summary  indicates  no  statistically  significant  difference  in 
ratings  assigned  to  those  who  had  simulator  training  and  those  who  did 
not. 
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Table  8 


Analysis  of  Variance  of 

First  Helicopter 

Flight 

Ratings 

Source 

SS 

df 

MS 

F 

P 

Between  Subjects 

10.90 

18 

Type  of  Training 

1.07 

1 

1.07 

00 

t— i 

NS 

Subjects  Within  Groups 

9.83 

17 

.58 

Within  Subjects 

12.  36 

76 

Maneuvers 

1.93 

4 

.48 

3.20 

<.05 

Type  Training 

x Maneuver 

.22 

4 

.06 

.40 

NS 

B x Subjects  Within  Groups 

10.  21 

68 

.15 

Total 

23.26 

94 

Tt  appears  that  cruise  might  have  a slightly  higher  rating  than 
the  other  maneuvers,  as  indicated  by  the  statistical  significance  of 
the  maneuver  effect. 

The  same  rationale  for  rating  the  first  helicopter  flight  was  ap- 
plied to  the  trial  at  which  the  criterion  performance  was  attained. 
Table  9 presents  a summary  of  the  split-plot  factorial  analysis  of 
variance  of  these  rating  data.  A least-squares  solution  was  also 
applied  to  these  data  to  adjust  for  unequal  subgroup  size. 

Once  again,  the  data  analysis  shows  no  significant  differences 
between  ratings  of  those  trained  in  the  helicopter  and  those  who  re- 
ceived prior  simulator  training.  Again,  there  was  a difference  in 
the  ratings  of  the  maneuvers,  indicating  that  cruise  received  a higher 
rating.  This  is  not  a surprising  result  because  cruise  flight  is 
probably  the  easiest  maneuver  to  perform. 

B'inally,  IP  ratings  having  been  obtained  for  the  last  simulator 
flight  on  each  of  the  maneuvers  and  on  the  first  and  last  helicopter 
flights  on  each  maneuver,  correlation  coefficients  were  computed  be- 
tween each  of  the  maneuvers  to  determine  the  trend  of  the  student 
pilot's  performance  in  the  helicopter.  Table  10  is  a matrix  showing 
the  correlation  of  rating  scores  of  the  maneuvers  in  the  simulator  and 
those  same  maneuvers  on  first  helicopter  flight. 
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Table  9 


ft 


Analysis  of  Variance  of  Pinal  Helicopter  IP  Ratings 


Source 

SS 

"'"1 

df  x 

MS 

F 

P 

Between  Subjects 

134.95 

18 

7.50 

Type  of  Training 

3.13 

1 

3.13 

.40 

NS 

Subjects  Within  Groups 

131.82 

17 

7.75 

Within  Subjects 

128.33 

76 

1.69 

Maneuvers 

26.  89 

4 

6.  72 

4.89 

.005 

Type  Training 

x Maneuver 

7.94 

4 

1.99 

1.45 

NS 

B x Subjects  Within  Groups 

93.50 

68 

1.38 

Total 

260.15 

94 

2.77 

Table  10 

Correlation  Between  Ratings  of  Last  Simulator  and 
First  Helicopter  Performance 

First  Helicopter  Flight 

Takeoff  Night  Night 

Simulator  & Climb  Cruise  Approach  Autorotation 

Night 
Circling 
Appro ach 

Takeoff 

& Climb  .03 


Cruise  -.14 

Night 

Approach  .45 

Night 

Autorotation  -.30 


Night 

Circling 

Approach 


-.27 


Each  correlation  coefficient  was  tested  with  Fisher's  t to  deter- 
mine whether  it  differed  significantly  from  zero,  the  expected  popula- 
tion correlation.  None  was  significant.  This  result  is  interpreted 
to  mean  that  no  rating  in  the  simulator  could  be  used  to  predict  reli- 
ably the  rating  on  the  first  helicopter  flight. 

In  a similar  manner,  coefficients  were  computed  for  the  last 
simulator  flight  and  the  last  helicopter  flight.  Table  11  presents 
the  results. 


Table  11 

Correlation  Between  Ratings  of  Last  Simulator  and 
Last  Helicopter  Performance 


Last  Helicopter  Flight 

Night 

Circling 

Approach 

Simulator 

Flight 

Takeoff  Night 

& Climb  Cruise  Approach 

Night 

Autorotation 

Takeoff 
& Climb 

.00 

Cruise 

.13 

Night 

Approach 

-0.06 

Night 

Autorotation 

-.56 

Night 

Circling 

Approach 

-.14 

Again,  using  Fisher's  t,  no  coefficient  differed  significantly 
from  zero. 

To  determine  if  trials  to  criterion  in  the  simulator  could  be  used 
to  predict  subsequent  trials  required  in  the  helicopter,  a correlation 
coefficient  was  computed  on  each  of  the  maneuvers.  Table  12  shows  the 
results.  None  of  the  correlation  coefficients  in  Table  12  was  statis- 
tically significant  from  zero. 
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Table  12 


Correlation  of  Simulator  Trials  to  Criterion 
and  Helicopter  Trials  to  Criterion 


Simulator 

Trial 

Helicopter  Trial 

Takeoff  Night  Night 

& Climb  Cruise  Approach  Autorotation 

Night 

Circling 

Approach 

Takeoff 
& C limb 

-.01 

Cruise 

-.09 

Night 

Approach 

.01 

Night 

Autorotation 

.01 

Night 

Circling 

Approach 

-.42 

On  balance,  the  results  of  this  evaluation  of  the  NVS/UH1FS  system 
did  not  demonstrate  reliable  effects  on  the  students  who  received 
training  in  it  when  their  performance  was  compared  to  matched  subjects 
with  no  simulator  training. 

An  analysis  of  variance  was  computed  on  trials  to  criterion  per- 
formance, as  indicated  previously,  and  no  significant  differences  be- 
tween simulator  and  helicopter-trained  students  were  detected.  In 
computing  correlation  coefficients,  however,  it  was  noted  that  those 
trained  in  the  simulator  exhibited  more  variability  in  trials-to- 
criterion  scores  in  the  simulator  than  in  the  aircraft  on  both  the 
night  approach  (6.68  to  1.47)  and  the  night  autorotation  (9.32  to 
2.09).  The  data  are  not  conclusive.  A corresponding  examination  of 
the  variability  of  those  two  maneuvers  for  those  with  helicopter 
training  showed  variances  of  .90  for  the  night  approach  and  .48  for 
the  autorotation.  Because  the  overall  tests  showed  no  reliable  dif- 
ferences, these  findings  were  not  pursued.  There  may  be  some  disrup- 
tion in  the  simulator  on  these  two  maneuvers  that  does  not  occur  in 
the  aircraft;  yet  it  was  not  of  sufficient  effect  to  influence  the 
later  transfer  performance.  This  effect  is  discussed  in  more  detail 
in  the  Discussion  section. 


Finally,  the  question  was  asked  whether  the  simulator  trials  to 
criterion  and  the  final  simulator  performance  rating  could  be  used  to 
predict  trials  to  criterion  in  the  transfer  task.  A multiple  correla- 
tion coefficient  was  computed  for  these  variables  on  the  takeoff  and 
climb,  cruise,  night  approach,  and  night  autorotation.  Table  13  pre- 
sents the  data  in  terms  of  the  coefficient  of  correlation,  the  coeffi- 
cient of  multiple  determination  for  the  four  maneuvers,  and  the  index 
of  forecasting  efficiency.  The  multiple  correlation  coefficient  (R) 
shows  the  relationship  of  the  two  predictors  to  the  criterion  task. 

The  coefficient  of  multiple  determination  (r2)  is  an  index  of  the 
amount  of  variance  in  the  transfer  trials  to  criterion  that  can  be 
accounted  for  by  the  variance  in  the  predictor  variables  taken  to- 
gether. The  index  of  forecasting  efficiency  (E)  is  the  percentage 
reduction  in  errors  of  prediction  by  reason  of  the  multiple  correla- 
tion. In  these  cases,  E indicates  in  the  case  of  the  night  approach, 
for  example,  that  predicting  the  score  of  night  approach  performance 
in  the  transfer  task  by  means  of  the  multiple  R is  on  the  order  of 
8%  better  than  using  a knowledge  of  the  mean  scores  of  the  night  ap- 
proach trials-to-criterion  data.  However,  as  shown  by  the  R2 , much 
of  the  variance  remains  unaccounted  for  by  the  relationship.  As  a 
consequence,  predicting  the  transfer  performance  on  these  four  tasks 
would  be  done  with  low  confidence  in  the  predicted  outcome. 


Table  13 


Multiple  Coefficient  of  Correlation,  Coefficient  cf  Multiple 
Determination,  and  Index  of  Forecasting  Efficiency 
In  Predicting  Transfer  Trials  to  Criterion  from 
Simulator  Trials  to  Criterion  and  Rating 


Maneuver 


Index 

Takeoff 
& Climb 

Cruise 

Night 

Approach 

Night 

Autorotation 

R 

.52 

.57 

.39 

.22 

R2 

.27 

.33 

.15 

.05 

E 

15% 

18% 

8% 

3% 
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Evaluation  III:  Informal  Evaluation  of  Instrument  Approach  Capabilit' 


Z 

To  assess  the  system's  ability  to  enhance  instrument  approach  to 
Standard  A lighting,  the  original  plan  was  to  develop  a series  of 
approaches  and  have  instrument  pilots  assess  the  suitability  of  the 
device.  To  this  end,  software  for  a front  and  back  course  instrument 
landing  system  (ILS)  approach  was  developed  for  Goldberg  stagefield,  as 
it  was  represented  on  the  NVS , and  an  approach  plate  was  drawn  to  show 
the  approaches. 

Circumstances,  time,  and  personnel  availability  prevented  a formal 
evaluation  of  this  capability.  Anecdotal  evidence  obtained  from  Flight 
Simulator  Division  personnel  and  other  aviator  personnel  indicated  that 
the  capability  to  execute  an  ILS  approach  and  breakout  at  several  ceil- 
ing heights  and  visibility  levels  was  satisfactory.  This  capability 
was  judged  to  be  suitable  as  an  adjunct  to  the  night  instrument  approach 
training.  There  were  limitations  to  this  capability,  but  they  dealt 
primarily  with  the  limitation  of  time  in  developing  the  software.  Simi- 
lar difficulties,  experienced  in  the  automated  tapes,  will  be  mentioned 
in  the  Discussion  section. 


In  this  further  discussion  of  the  total  system  evaluation,  topics 
of  a general  nature  are  treated,  as  they  influenced  the  evaluation. 

Additional  discussions  are  developed  concerning  the  two  major  evaluations. 

Although  this  evaluation  was  originally  designed  to  be  a suitability 
evaluation  of  the  NVS,  certain  factors  arose  that  tended  to  expand  the 

scope  of  the  effort.  As  a result  of  adding  the  visual  display,  many  , 

discrepancies  in  the  handling  characteristics  and  instrument  readouts 
of  the  UH1FS  were  noted;  obviously,  attaching  the  NVS  placed  on  the  system 
requirements  that  did  not  exist  for  an  instruments-only  simulator.  This 
led  to  several  problems,  for  example,  the  requirement  to  develop  the 
hover  mode.  Although  simulated  hover  in  the  UH1FS  for  instrument  training 
was  unnecessary,  it  is  a prime  requirement  for  contact  flight.  Similar 
problems  were  encountered  during  the  software  integration  and  some  fixes 
were  effected.  There  were  sufficient  handling  qualities  problems  to 
warrant  a delay  in  the  original  ready-for- training  date. 


The  four  instructor  pilots  were  diligent  in  working  with  Singer 
engineers  to  correct  deficiencies  caused  by  adding  the  visual  display. 
Some  examples  of  difficulties  follow,  to  give  perspective  to  the  problem 
of  integrating  the  NVS . 
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One  of  the  first  difficulties  was  noted  in  the  pilots'  complaints 
about  the  "sensitive"  collection  pitch  and  the  "bouncing"  of  the  helicopter 
when  attempting  to  land.  This  was  characterized  as  a "trampoline"  effect, 
caused  by  excessive  flexion  programed  into  the  cross  tubes. 

The  helicopter  would  also  draft  up  and  down.  The  draft  was  slow 
but  noticeable  and  required  constant  adjustment  of  collective  pitch. 

This  was  solved  by  increasing  stability  and  dampening  the  aircraft  when 
in  ground  effect.  However,  this  led  to  additional  problems,  namely  an 
increase  in  acceleration,  which  caused  a "float"  when  making  an  approach 
or  autorotatxon.  A solution  was  to  devise  an  acceleration  parameter  for 
takeoffs  and  a separate  deceleration  parameter  for  vertical  descent. 
However,  when  these  changes  were  made,  the  computer  approached  capacity 
and  no  additional  corrections  in  those  parameters  could  be  made . 

Additional  problems  in  RPM  needles  in  autorotation  led  to  other 
adjustments,  either  in  the  software  or  on  the  part  of  the  pilots  who  had 
to  accommodate  to  the  changes. 

Because  of  the  limited  field  of  view,  approaches  tended  to  be 
shallower  than  normal  throughout  the  evaluation.  At  the  beginning, 
the  IP's  identified  this  as  a problem  of  sufficient  magnitude  that  a 
centerline  had  to  be  constructed,  edge  markings  enhanced,  and  six  large 
stripes  added  to  the  lane. 

Altitude  judgments  were  difficult,  at  low  altitudes  in  particular. 

A tower  was  added  to  the  stagefield  to  aid  in  altitude  judgment  for 
approaches  and  helped  solve  problems  in  altitude  judgment  and  rate  of 
closure  judgments. 

One  particular  problem  to  which  the  pilots  had  to  accommodate  was 
hover  altitude.  Although  the  eye  height  was  calculated  to  be  correct, 
perceptually  it  gave  the  appearance  of  being  too  high.  Attempts  to  adjust 
this  led  to  a kind  of  de- focus;  the  IP's  decided  they  would  rather  have 
the  simulation  in  focus  at  a higher  altitude  and  explain  it  as  a training 
problem  if  a question  arose. 

The  confined  area  had  to  be  modified.  On  approach,  the  horizon 
would  be  lost  near  30  ft.  AGL;  then,  because  of  the  uniformity  of  the 
visual  scene,  disorientation  and  a hard  landing  would  usually  result. 

Two  vertical  rows  of  lights  were  added  to  the  barrier,  and  a rear  barrier 
was  also  added  to  the  scene  to  assist  in  judging  rate  of  closure. 

Additional  recommendations  and  improvements  could  have  been  made — 
but  those  actually  made  were  required,  in  the  estimation  of  the  IP’s,  to 
reach  a minimal  acceptable  level  for  training. 
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In  addition  to  the  necessity  for  the  fixes  described  above,  con- 
siderable difficulty  was  noted  with  NVS  hardware  reliability  in  the 
beginning  portion  of  the  evaluation.  The  system  was  released  by  Singer 
for  the  test  on  27  June  1976,  but  through  the  first  2 weeks,  training 
could  only  be  attempted  and  was  seldom  completed.  Consequently,  training 
ceased,  and  Singer  was  requested  to  make  repairs.  The  suitability  test 
was  begun  again  on  19  July,  and  with  minor  exceptions  the  reliability 
improved  considerably.  A complete  NVS  system  maintenance  log  was  kept 
by  Singer  from  1 May  1976  through  9 December  1976,  describing  each  failure 
and  the  corrective  action  taken.  During  the  period  of  the  evaluation, 
there  were  32  recorded  failures,  caused  primarily  by  electronic  component 
malfunctions . 


Instructor  Pilot  Evaluation 


After  the  problems  had  been  correctec.,  the  IP's  who  had  spent  much 
time  in  the  simulator  were  asked  to  conduct  their  evaluations.  The 
results  of  these  evaluations  did  conclude  that  the  device  was  suitable 
for  continued  training,  but  several  points  merit  further  discussion. 

The  rating  scale  used  by  the  IP's  was  developed  with  the  intent  of 
evaluating  the  device  as  a contact  flight  training  device.  As  mentioned 
previously,  the  maneuver  element  descriptions  were  not  designed  to  address 
visual  factors  only.  Nevertheless,  unless  some  pilot  opinions  and  comments 
obtained  after  the  evaluation  are  taken  into  account,  some  effects  (or 
lack  of  effect)  of  the  visual  system  are  obscured. 

Although  hovering  and  other  near-ground  maneuvers  were  accomplished, 
one  pilot  felt  that  without  lights  to  the  front  of  the  helicopter  at  a 
hover,  altitude  could  not  be  maintained;  this  was  attributed  to  the 
restricted  field  of  view.  This  commend  was  typical  and  reflected  a 
consensus  of  the  pilots  in  that  ground  track  was  consistently  rated  low. 
Thus,  although  a maneuver  was  rated  as  acceptable,  in  those  cases  in  which 
the  cues  were  from  out  the  window,  the  pilots  felt  the  cues  were  insuffi- 
cient. For  example,  if  a takeoff  and  climb  were  rated  high,  it  was 
because  of  the  overall  device  capability,  since  the  heading  could  be 
maintained  via  the  RMI  or  other  instruments.  Indred,  this  was  the 
method  required  for  many  of  the  maneuvers. 

The  IP's  found  that  not  all  maneuvers  were  simulated  equally  well. 

In  addition  to  those  maneuvers  that  simulated  the  aircraft  with  reasonable 
fidelity,  the  hydraulic  power  failure  was  simulated,  but  with  low  fidelity. 
That  is,  its  response  was  unlike  the  response  after  a hydraulic  power 
failure  in  the  aircraft.  Their  evaluations  further  revealed  that  rear- 
ward hovering  flight  was  not  programed. 
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From  the  IP  evaluation  sheets,  it  is  evident  that  the  out-the-window 
display  was  less  than  adequate  when  required  to  provide  cues  such  as  the 
ground  track,  terrain  for  high  or  low  reconnaissance,  and  maneuvers 
related  to  the  pinnacle  operation.  Also,  as  pointed  out  previously,  the 
restricted  visual  angle  required  a shallower  approach  than  the  pilots 
desired.  Pilot  comments  indicated  that  the  side  window  and  chin  windows 
in  the  helicopter  were  more  useful  to  them  than  they  had  first  realized, 
and  lack  of  such  displays  caused  an  alteration  in  their  instruction. 

Finally,  a factor  that  cannot  be  assessed  but  should  be  mentioned 
is  the  attitude  with  which  the  instructor  pilots  approached  the  task. 

At  the  outset  they  were  somewhat  antisimulation  and  expressed  some 
concern  about  the  amount  of  time  required  away  from  their  duty  assign- 
ments, but  as  they  became  involved  in  the  process,  they  developed  a 
more  prosimulation  attitude.  There  was  a general  expression  of  confi- 
dence in  their  work  and  a willingness  to  work  longer  hours  to  make  the 
evaluation  as  successful  as  possible.  This  is  to  say  that  a bias  would 
be  hard  to  document,  because  what  may  have  started  as  a bias  against  the 
device  seems  to  have  shifted  to  an  extent  that  a leniency  could  have 
been  reflected  in  the  evaluation  ratings.  The  criteria  established  by 
the  instructors  in  rating  the  device  seem  to  have  been  shared  by  all,  as 
evidenced  by  the  coefficient  of  concordance,  but  it  is  still  not  clear 
as  to  why  the  large  number  of  high  ratings  resulted.  The  reasons  could 
only  be  conjectured. 


Transfer  of  Training  Evaluation 

The  device  was  hypothesized  to  render  a positive  transfer  effect. 

It  did  not,  so  further  discussion  is  warranted. 

Hampering  of  the  transfer  study  at  the  outset  by  the  equipment 
reliability  has  been  discussed.  But  problems  also  arose  in  attempting 
to  integrate  the  evaluation  into  the  existing  Aviation  Center  training 
program. 

As  has  been  mentioned,  it  was  not  possible  to  assign  students  to 
the  evaluation,  and  consequently  they  were  obtained  on  a noninterference 
basis.  This  difficulty,  along  with  the  shortage  of  time  for  the  evaluation, 
had  significant  influence  on  the  evaluation.  The  evaluation  had  an 
absolute  deadline,  after  which  the  instructor  pilots  were  to  revert  to 
their  regular  duties.  Becuase  of  a concern  to  obtain  and  train  as  many 
subjects  as  possible,  the  method  of  grading  performance  was  influenced. 


Criterion  performance  had  been  discussed  at  length  with  the 
instructors,  and  they  had  agreed  that  two  consecutive  trials  performed 
within  acceptable  limits,  as  defined  by  the  grade  sheet,  would  be  a 
sufficient  indicator  of  success.  But  with  this  as  the  referent,  there 
was  perhaps  an  insufficient  capability  to  discriminate  performance. 

More  precise  measures  of  discrimination  might  have  demonstrated  different 
results  in  the  transfer  study. 

The  amount  of  variability  in  the  simulator  training  of  students, 
which  did  not  appear  in  the  transfer  task,  could  mean  that  there  was 
some  cue  or  activity  in  the  simulator  that  disrupted  performance  in 
the  simulator  but  was  not  of  sufficient  magnitude  or  duration  to  affect 
helicopter  performance.  On  the  other  hand,  such  an  effect  could  also 
be  a result  of  lack  of  discrimination  of  performance  by  the  instructors 
in  the  transfer  task.  At  least  one  reason  could  be  the  lack  of  time  for 
accomplishing  that  transfer  task. 

As  mentioned  previously,  Thursday  and  Friday  of  each  week  were  used 
for  this  training.  The  press  of  trying  to  train  six  to  eight  students 
to  criterion  on  five  maneuvers  and  return  aircraft  by  scheduled  return 
times  probably  led  to  discriminations  that  were  less  fine  than  usual,  by 
otherwise  dedicated  instructors. 

A related  factor  that  increased  the  workload  was  the  illness  and 
loss  of  one  instructor,  with  no  replacement  for  the  majority  of  the 
evaluation . 

One  factor  that  undoubtedly  influenced  the  outcome  of  the  study  was 
the  point  in  the  student  flow  at  which  the  students  were  obtained;  addi- 
tional study  with  the  NVS  might  show  a more  appropriate  time.  The  student 
pilots  were  obtained  during  the  final  stages  of  instrument  training.  They 
had  not  received  any  contact  training  in  the  UH-.l . Furthermore,  there  is 
no  requirement  for  the  instrument  instructor  pilots  to  qualify  these 
students  in  any  maneuvers.  Nevertheless,  some  usually  teach  their 
students  to  fly  some  contact  maneuvers,  including  landings,  while  others 
do  not.  Thus,  there  was  no  positive  control  over  the  entire  amount  and 
type  of  experience  accrued  by  the  students.  Obviously,  other  situations 
could  yield  a more  appropriate  test  of  the  NVS.  Such  a reevaluation  may 
be  needed  to  make  definite  conclusions  concerning  the  results.  In 
addition,  some  time  must  be  made  available  to  explore  the  configuration 
of  the  NVS  and  to  select  a more  appropriate  time  for  undertaking  the 
training.  If  these  latter  factors  are  remedied,  an  evaluation  yielding 
conclusive  recommendations  could  be  made . 
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Miscellaneous 


As  time  permitted,  experienced  aviators  were  invited  to  fly  the 
system  and  their  opinions  were  solicited.  Almost  without  exception, 
those  individuals  who  were  older,  who  were  not  in  flying  assignments, 
or  who  had  not  flown  in  some  time,  reported  nausea — in  the  words  of 
one  person,  "I  was  deathly  ill."  This  response  is  not  uncommon,  and 
previous  reports  have  hinted  that  a possible  cause  is  a lag  in  the 
onset  of  the  visual  system  after  the  motion  system  response.  These 
past  experiences  and  the  reasons  for  them  indicate  a need  to  measure 
the  visual/motion  system  to  determine  whether  a lag  exists  and,  if  it 
does,  whether  it  is  of  sufficient  magnitude  to  cause  performance 
decrements . 
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High  reconnaissance  completed  (minimum  200  feet  AHO) . 

Can  perfectly  perform  circling  approach. 

Low  reconnaissance  completed. 

Takeoff  gas-producer  N.  not  to  exceed  go-no-go  limits. 


APPENDIX  B 


STUDENT  GRADE  SHEET 
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I NIGHT  APPROACH! 
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